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Designing  Selection  Tests  for  the  Future 
National  Airspace  System  Architecture 


What  abilities  will  be  required  of  controllers  to 
operate  and  maintain  the  air  traffic  control/manage- 
ment  (ATC/M)  systems  of  the  future?  Cole  suggested 
that  the  ATC/M  systems  of  the  future  “will  require 
individuals  with  a  different  mix  of  abilities  than  what 
is  needed  today”  (1995,  p.  47).  Similarly,  the  Na¬ 
tional  Research  Council  recently  suggested  that  dif¬ 
ferent  abilities,  or  a  different  weighting  of  abilities, 
might  be  required  under  different  forms  of  future 
automation  (Wickens,  Mavor,  &  McGee,  1997,  p. 
68).  However,  empirical  data  are  presently  lacking  to 
describe  the  mix  of  abilities  required  to  operate  and 
maintain  the  emerging  National  Airspace  System 
(NAS)  architecture,  as  described  in  National  Air- 
space  Architecture  version  2. 0  (FAA,  1996).  The  goal  of 
the  Civil  Aeromedical  Institute  (CAMI)  Training  and 
Organizational  Research  Laboratory’s  future  selec¬ 
tion  research  program  is  to  develop  scientific  tools 
and  collect  data  to  describe  and  assess  the  mix  of 
abilities  likely  to  be  required  of  future  controllers, 
technicians,  and  systems  specialists. 

This  development  of  the  tools  and  data  to  identify 
and  assess  abilities  required  to  operate  and  maintain 
the  future  NAS  architecture  is  needed  to  maintain  a 
high  quality  workforce  in  the  FAA’s  safety- related 
occupations.  The  development  of  these  tools  will 
occur  within  the  context  of  three  important  trends. 
First,  as  noted  in  the  National  Airpsace  Architecture 
version  2.0^  “Personnel  costs  are  the  main  drivers  of 
O&M  (Operations  and  Maintenance)  funding”  (p.  2- 
89) .  For  example,  personnel  costs  account  for  the  bulk 
of  expected  O&M  funding  from  1998  through  2015 
(  Figure  2. 2. 4-3,  “Proposed  architecture  O&M  costs 
and  expected  available  O&M  funding,”  p.  2-11). 
Personnel  will  account  for  almost  100%  of  the  avail¬ 
able  Office  of  Management  and  Budget  (OMB)  ex¬ 
pected  passback  of  about  $4  billion  for  O&M  by  the 
year  2015,  according  to  the  NAS  architecture  docu¬ 
ment.  Cumulative  personnel  costs  between  1998  and 


2015  may  well  exceed  $50  billion,  dwarfing  total 
investment  in  procuring  the  hardware,  software,  and 
services  comprising  the  NAS.  Putting  the  right  people 
into  the  jobs  of  using,  operating,  and  maintaining  the 
NAS  through  efficient  and  effective  personnel  selec¬ 
tion  is  one  relatively  inexpensive  management  tool  for 
maximizing  the  agency’s  return  on  what  may  well  be 
a  significant  national  investment. 

Second,  there  will  be  a  generational  change  in  the 
ranks  of  controllers,  technicians,  and  systems  special¬ 
ists  as  current  employees  reach  retirement  eligibility 
and  are  replaced  by  a  new  generation.  That  process  has 
already  started  in  the  ranks  of  field  maintenance 
technicians,  as  shown  in  Figure  1 .  By  the  year  2002, 
according  to  available  Airway  Facilities  demographic 
data  (Airway  Facilities,  1992),  about  46%  of  the 
current  field  maintenance  workforce  will  be  eligible 
for  retirement;  reasonably  one  would  expect  an  even 
greater  proportion  of  the  current  generation  to  reach 
retirement  eligibility  beyond  2002.  Even  with  staffing 
reductions  and  reorganization,  significant  numbers  of 
new  technicians  will  likely  be  required.  This  demo¬ 
graphic  trend  is  echoed  in  the  ranks  of  controllers,  as 
the  post-strike  generation  begins  to  reach  retirement 
eligibility  starting  around  2005  (Figure  1).  By  the  year 
2020,  approximately  80%  of  the  post-strike  genera¬ 
tion  of  controllers  will  be  eligible  for  retirement.  As  a 
consequence,  the  agency  will  need  reasonable  and  job- 
related  tools  to  select  the  few  from  among  the  many 
applicants  who  have  the  “right  stuff’  to  become  the 
operators,  maintainers,  and  managers  of  the  emerging 
NAS  architecture. 

Third,  new  ATC/M  technological  innovations  will 
be  entering  service  in  the  NAS  at  about  the  same  time 
this  generational  change  is  occuring,  as  shown  in 
Figure  2.  For  example,  the  consolidation  of  Airway 
Facilities  Maintenance  Control  Centers  (MCCs)  into 
9  Operations  Control  Centers  (OCCs)  under  the 
National  Infrastructure  Management  Systems  (NIMS) 
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Figure  2 

ATC/M  technology  programs  overlaid 
on  top  of  FAA  technical  workforce  retirement  eligibility  trends 
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Figure  1 

Estimated  proportion  of  FAA 
technical  workforce  reaching  retirement  eligibility 
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project  will  occur  just  as  the  proportion  of  retirement 
eligible  field  technicians  reaches  about  46%.  A  new 
ATC/M  infrastructure  will  be  developed  through  the 
ARTCC  Display  System  Replacement  (DSR)  and 
terminal  Standard  Tower  Automation  Replacement 
System  (STARS)  programs  through  the  turn  of  the 
century.  New  ATC/M  decision  support  systems  (DSS) 
will  be  developed  during  the  same  time  period,  as  the 
post-strike  generation  of  controllers  continues  to  age. 
Controller  retirement  eligibility  will  begin  to  rise  at 
about  2005,  just  as  these  new  DSSs  are  supposed  to  be 
integrated  with  the  DSR/ STARS  infrastructure  (Fig¬ 
ure  2).  The  operational  concept  known  as  “Free  Flight” 
will  frame  these  technological  changes.  Selecting  the 
next  generation  of  controllers  and  systems  specialists 
to  operate  and  manage  the  new  ATC/M  systems  in  the 
Free  Flight  environment  will  be  an  important  man¬ 
agement  task  that  will  have  profound  effects  on  the 
safety  and  efficiency  of  the  future  NAS.  The  tools  and 
processes  to  be  used  by  agency  managers  to  select  the 
future  NAS  workforce  must  reflect  the  human  ability 
and  performance  requirements  of  the  emerging  AT  C/ 
M  architecture. 

These  three  trends  —  the  overwhelming  cost  of 
staffing  the  NAS,  the  inevitable  generational  changes 
in  controller  and  technician  workforces,  and  the  con¬ 
tinuing  introduction  of  new  ATC/M  technologies  to 
support  the  “Free  Flight”  concept  of  operations  — 
define  a  research  requirement  to  identify  the  person¬ 
nel  selection  requirements  associated  with  the  evolv¬ 
ing  NAS  architecture.  That  is,  these  inescapable  facts 
of  cost,  generational  change,  and  technological  inno¬ 
vation,  force  the  agency  to  confront  the  question, 
“What  is  the  mix  of  abilities  required  to  operate  and 
maintain  the  emerging  NAS  architecture?”  The  Civil 
Aeromedical  Institute’s  (CAMI)  Training  and  Orga¬ 
nizational  Research  Laboratory  research  program  on 
future  selection  proposes  to  address  these  issues  in 
three  phases. 

First,  the  FAA’s  research  program  will  develop  a 
baseline  profile  describing  the  abilities  required  to 
use,  operate,  and  maintain  the  current  NAS.  Second, 
the  research  program  will  develop  and  apply  scientific 
tools  to  identify  changes  in  selection  requirements  in 
parallel  with  ATC/M  systems  development.  Third, 
the  research  program  will  develop,  validate,  and  de¬ 


liver  new  personnel  selection  technologies  to  reflect 
the  human  ability  and  performance  requirements  of 
the  emerging  NAS  architecture. 

Current  baseline 

A  baseline  profile  of  abilities  required  of  controllers 
to  operate  the  current  NAS  was  developed  by  the  job 
analysis  phase  of  the  Separation  and  Control  Hiring 
Assessment  (SACHA)  procurement  in  late  1 994  (Nick¬ 
els,  Bobko,  Blair,  Sands,  &  Tartak,  1995).  That 
analysis  of  the  terminal,  en  route,  and  flight  service  air 
traffic  control  specialist  (ATCS;  GS-2152)  job  de¬ 
scribed  baseline  en  route  Host  Computer  System 
(HCS),  terminal  Area  Radar  Terminal  Service  (ARTS), 
and  Model  I  Automated  Flight  Service  Station  (AFSS) 
job  duties  and  functions.  The  SACHA  job  analysis 
also  identified  the  abilities  required  for  successful 
performance  in  each  environment. 

Development  of  a  baseline  profile  of  Electronic 
Technician  (ET;  FG-0856)  and  Transportation  Sys¬ 
tems  Specialist  (TSS;  FG-2101)  job  duties,  functions, 
ands  abilities  has  been  proposed  for  FY  1997  using 
FAA  research,  engineering,  and  development  (RE&D) 
human  factors  funds.  Booz,  Allen,  Hamilton  con¬ 
ducted  an  analysis  of  the  ET  knowledges  and  skills 
required  to  maintain  specific  NAS  equipment  and 
systems  in  1993.  However,  that  analysis  did  not 
describe  the  ET  or  TSS  jobs  as  a  whole  in  a  way 
technically  or  legally  suitable  to  support  the  develop¬ 
ment  of  personnel  selection  procedures  under  the 
Uniform  Guidelines  on  Employee  Selection  Proce¬ 
dures  (29  CFR  1607).  A  new  baseline  analysis  will 
provide  a  starting  point  for  identifying  changes  in,  or 
new,  ability  and  performance  requirements  associated 
with  programs  such  as  the  National  Infrastructure 
Management  System  (NIMS). 

Identifying  changes 

The  next  element  of  CAMI’s  Training  and  Organi¬ 
zational  Research  Laboratory  selection  program  is  to 
identify  changes  in  job  duties  and  human  perfor¬ 
mance  requirements.  The  continuing  research,  devel¬ 
opment,  and  introduction  of  new  technologies  into 
the  ATC/M  environment  creates  an  impression  of 
rapid  change  in  the  job.  Some  technologies  being 
introduced,  such  as  the  planned  Display  System  Re- 
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placement  (DSR)  seem  largely  incremental  in  nature, 
and  appear  unlikely  to  dramatically  change  the  na¬ 
ture,  structure,  or  activities  of  the  controller’s  job. 
Other  technologies,  such  as  the  User  Request  Evalu¬ 
ation  Tool  (URET),  Center-Terminal  Automation 
System  (CTAS),  Automated  Enroute  ATC  (AERA), 
and  other  decision  support  systems  may  substantially 
change  the  nature,  structure,  and  activities  of  the 
controller  job.  The  development  and  implementation 
of  the  Operational  Control  Center  (OCC)  concept 
for  management  of  the  NAS  infrastructure  may  simi¬ 
larly  have  a  profound  effect  on  the  nature,  structure, 
and  ability  requirements  of  the  technician  and  system 
specialist  job. 

Given  the  2  to  3  year  lead  time  between  selection 
and  certification  of  a  person  as  a  full  performance  level 
(FPL)  controller,  technician,  or  systems  specialist,  the 
agency  cannot  wait  for  these  new  ATC/M  technolo¬ 
gies  to  emerge  from  the  RE&D  cycle  to  develop  and 
field  new  selection  tests.  FAA  tests  should  be  fielded 
prior  to  implementation  of  the  new  technologies  to 
assure  a  flow  of  qualified  personnel  into  the  training 
and  development  pipeline.  The  analyses  to  identify 
changes  in,  or  new,  predictors  and  criteria  should  be 
conducted  in  parallel  with  systems  development. 

To  support  the  early  development  of  new  selection 
instruments,  a  methodology  for  analyzing  the  selec¬ 
tion-oriented  requirements  of  ATC/M  systems  in 
various  stages  of  the  acquisition  cycle  is  needed.  As 
one  might  expect,  the  type,  amount,  and  quality  of 
information  available  about  a  proposed  system  varies 
directly  with  the  system’s  developmental  stage.  For 
example,  the  only  information  available  early  in  the 
acquisition  cycle  might  be  a  high-level  requirements 
analysis  and  the  relatively  global  concept  of  opera¬ 
tions  for  fulfilling  those  requirements.  Specific  tech¬ 
nological  solutions  might  be  developed  in  successive 
iterations.  Information  available  with  each  iteration 
of  the  systems  design  process  might  include  more 
detailed  functional  requirements,  data  specifications, 
human-computer  interface  and  interaction  specifica¬ 
tions,  system  integration  plans,  and  implementation/ 
transition  plans.  Cognitive  and  other  task  analyses 
may  be  developed  as  part  of  the  iterative  systems 
development  cycle.  It  may  be  possible  to  develop  a 
model  of  how  a  proposed  ATC/M  system  will  be 


incorporated  into  the  controller,  technician,  or  sys¬ 
tem  specialist  job  on  the  basis  of  these  documents  as 
the  system  moves  through  the  development  cycle. 

Practical  methods  do  not  currently  exist  in  the 
selection  psychologist’s  tool  kit  for  analyzing  systems 
under  development  to  forecast  their  human  KSAO 
requirements.  Most,  if  not  all,  currently  available  job 
analysis  techniques  used  in  applied  settings  to  support 
personnel  selection  test  development  and  validation 
are  intended  for  use  with  relatively  stable  jobs  that 
incumbents  are  already  performing.  Available  meth¬ 
ods  are  ill-suited  for  analysis  of  the  requirements  of 
new  systems  (Schneider  &  Konz,  1989).  Methods 
such  as  “strategic  job  analysis”  (Schneider  &  Konz) 
have  not  found  wide  application  because  they  have 
been  difficult  to  use  in  applied  settings.  Also,  they 
may  yield  less  information  than  is  desirable  for  de¬ 
signing  new  selection  systems  (Manning  &  Broach, 
1992).  Factors  such  as  the  amount  of  information 
available  about  the  new  ATC/M  system  and 
compositon  of  subject  matter  expert  (SME)  panels 
appear  to  significantly  influence  the  results  obtained 
from  a  “strategic  job  analysis.” 

Traditional  job  analysis  methods  are  focused  on 
discrete  tasks,  rather  than  on  business  processes  as  the 
unit  of  analysis  (Cascio,  1995).  Moreover,  traditional 
job  analysis  methods  do  not  typically  take  into  ac¬ 
count  team  work  requirements  (Salas,  Dickinson, 
Converse,  &  Tannenbaum,  1992;  see  also  Dieterly 
(1988)  for  a  review  on  team-oriented  job  analysis). 
Traditional  job  analysis  techniques  are  also  poorly 
suited  to  the  description  of  the  covert,  cognitive,  or 
information  processing-based  elements  of  jobs  (Hogan, 
Broach,  &  Salas,  1991).  Finally,  job  analysis  has 
typically  been  a  “one-shot”  affair.  Rarely,  if  ever,  are 
job  requirements  evaluated  on  an  iterative  basis  to 
determine  if  there  are  any  important  changes  in  either 
the  profile  of  abilities  underlying  successful  perfor¬ 
mance  or  the  definition  of  successful  performance. 
The  “shelflife”  of  a  competently  done  selection  test 
validation  study  is  about  five  years  (Lefkowitz  & 
Gebbia,  1995),  suggesting  a  requirement  to  reevaluate 
job  requirements  on  an  iterative  basis.  A  job  analytic 
methodology  that  can  be  used  iteratively  in  parallel 
with  ATC/M  system  design,  development,  and  opera¬ 
tional  evaluation,  is  required  to  meet  the  three  chal- 
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lenges  of  cost  management,  generational  change,  and 
new  technologies.  This  methodology  should  identify 
changes  in,  and  new,  controller  KSAO  requirements 
as  the  basis  for  future  selection  systems.  Such  an 
analytic  methodology  would  take  into  account  the  con¬ 
tinuous  process  nature  of  the  controller  and  systems 
specialist  functions  as  well  as  team  work  requirements. 

CAMI  initiated  the  first  phase  of  the  development 
of  a  tool  or  methodology  for  strategic  job  analysis  at 
the  beginning  of  FY  1997  (see  Caliber  Associates, 
1996).  The  prototype  methodology  is  expected  to  be 
available  by  the  end  of  FY  1997.  The  selection  re¬ 
search  program  then  calls  for  testing  and  refining  the 
strategic  job  analysis  tool  on  selected  systems  at  vari¬ 
ous  stages  in  the  development  cycle  (concept,  proto¬ 
type,  and  operational  test  and  evaluation)  in  FY98. 
The  goal  is  to  then  apply  the  methodology  to  other 
systems  described  in  the  National  Airspace  System 
Architecture  version  2.0  (FAA,  1996),  in  parallel  with 
the  evolution  of  the  NAS.  From  those  iterative  analy¬ 
ses,  new,  changed,  and  continuing  selection  require¬ 
ments  can  be  identified  to  serve  as  the  basis  for  the 
evolution  of  agency  recruitment  and  selection  programs. 

Developing  future  tests 

The  final  major  component  of  the  FAA’s  selection 
research  program  is  the  development  and  validation 
of  the  actual  tests,  criteria,  and  procedures  for  select¬ 
ing  the  next  generation  of  controllers,  technicians, 
and  systems  specialists.  Flexible,  living,  and  dynamic, 
data-driven  recruitment  and  selection  systems  are 
envisioned,  rather  than  the  static,  monolithic  pro¬ 
cesses  of  the  past.  New  tests  will  capitalize  on  the  open 
systems  architecture  and  multimedia  capabilities  of 
personal  computers  and  networks.  New  testing  tech¬ 
nologies  currently  under  development  by  CAMI 
include  (a)  variable-item  generation  for  knowledge  and 
skill-based  tests,  (b)  simulation-based  assessment  of  abil¬ 
ity,  and  (c)  simulation-based  assessments  of  team  work. 

The  research  on  variable-item  generation  addresses 
the  issue  of  test  security.  The  objective  is  to  develop  a 
process  to  generate  multiple  versions  of  any  single  test 
item,  within  defined  parameters,  in  such  a  way  that 
each  examinee  is  presented  with  a  unique  item.  This 
technology  can  be  coupled  with  computerized  item 
banking  and  scoring  to  support  computer-adaptive 


testing.  In  computer-adaptive  testing,  items  are  drawn 
from  a  pre-calibrated  pool  in  just  sufficient  numbers 
to  accurately  assess  the  candidate’s  knowledge  or  skill 
in  a  specified  domain.  Rather  than  all  examinees 
getting  the  exact  same  number  of  items,  some  may  get 
more  or  fewer  items,  based  on  mathematical  algo¬ 
rithms  for  estimating  ability  from  item  responses.  A 
test  using  both  technologies  would  present  each  ex¬ 
aminee  with  the  minimum  number  of  unique 
examplars  of  a  class  of  items  just  sufficient  to  accu¬ 
rately  assess  the  candidate’s  knowledge  in  that  area. 
The  end  product  of  the  technology  development  is  a 
secure,  cost-  and  time-efficient  test. 

The  second  testing  technology  under  development 
by  the  FAA  is  based  on  the  use  of  work  samples  as 
simulation-based  assessments  of  abilities.  The  first 
generation  of  simulation-based  tests  for  air  traffic 
control  selection  included  the  Multiplex  Controller 
Aptitude  Test  (MCAT)  and  the  FAA  Academy  ATCS 
Nonradar  Screen,  The  second  generation  of  perfor¬ 
mance-based  aptitude  testing  was  represented  by  the 
computerized  work  sample  known  as  the  Air  Traffic 
Scenario  Test  (ATST;  Broach  &  Brecht-Clark,  1993). 
Issues  of  measurement  reliability,  construct  represen¬ 
tation  and  nomothetic  span  in  complex  work  samples 
such  as  the  ATST  need  to  be  researched  to  draw 
meaningful  inferences  from  scores  generated  by  this 
class  of  performance-based  tests.  Other  research  issues 
that  need  to  be  addressed  include  the  identification  of 
what  aspects  of  the  job  are  to  be  represented  in  the 
work  sample  or  simulation,  the  influences  of  previous 
computer  and  video  game  experience,  training,  and 
performance  strategies. 

The  third  testing  technology  under  development 
by  CAMI  is  based  on  the  use  of  simulation  to  assess 
teamwork.  Teamwork  has  only  recently  been  recog¬ 
nized  as  an  important  aspect  of  human  performance 
in  the  NAS  (FAA,  1995).  For  example,  coordination 
was  cited  as  a  causal  factor  in  15%  of  1,038  low-to- 
moderate  severity  operational  errors  in  the  NAS  be¬ 
tween  1988  and  1991  (Rodgers  &  Nye,  1994). 
Coordination  errors  were  cited  as  causal  factors  in 
operational  errors  of  greater  severity  more  often  than 
other  causal  factors  such  as  misuse  of  displayed  radar 
data  (Rodgers  &  Nye).  A  recent  evaluation  of  opera¬ 
tional  errors  across  the  2 1  ART CCs  in  the  contintental 
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United  States  pointed  toward  other  teamwork  failures 
as  contributing  factors  in  the  recent  operational  error 
rate  trends  (Kirk,  Mayberry,  &  Lesko,  1 996;  Mayberry, 
Kropp,  Kirk,  Breitler,  &  Wei,  1995). 

There  is  a  significant  gap  in  the  general  and  ATC/ 
M  human  factors  scientific  literature  and  data  con¬ 
cerning  team  behaviors  and  their  impact  on  system 
safety  and  efficiency.  Moreover,  the  team  or  group 
performance  literature  is  both  large  and  fragmented, 
with  many  idosyncratic  definitions  and  operational 
representations  of  “team,”  “team  performance,”  and 
“teamwork”  (Driskell,  Hogan,  &  Salas,  1987;  Driskell, 
Salas,  &Hogan,  1987).  Team  work  is  a  very  active  area 
of  scientific  research,  with  no  standardized  methodol¬ 
ogy  for  the  assessment  of  behaviors.  Many  of  the 
assessment  instruments  currently  available  to  assess 
“team  work”  (however  defined)  were  developed  more 
on  the  basis  of  the  researcher^s  intuition,  rather  than 
formal  empirical  analysis  (Michaelson  &  Baker,  1 995) . 
The  majority  of  studies  in  the  “team”  literature  focus 
on  describing  teams  and  attitudes  toward  teams  rather 
than  behavior  within  the  team  (Michaelson  &  Baker) . 

The  focus  of  CAMFs  Controller  Teamwork  Evalu¬ 
ation  and  Assessment  Methodology  (CTEAM)  re¬ 
search  task  is  to  develop  a  prototype  computerized 
tool  for  the  assessment  of  team  work  behaviors  (Bailey, 
Broach,  6c  Enos,  1997).  CTEAM  provides  a  simu¬ 
lated  multi-player  radar-based  AT C  environment  that 
requires  cooperation  and  coordination  between  play¬ 
ers  to  accomplish  the  goal  of  getting  the  aircraft 
within  the  scenario  to  their  destinations  safely  and 
efficiently.  However,  that  simulated  multi-player  en¬ 
vironment  also  may  provide  a  basis  for  constructing  a 
process  or  procedure  for  the  selection  of  persons  who 
work  well  in  teams.  Moreover,  the  general  model  of 
CTEAM  may  be  translated  into  the  collaborative, 
shared  decision-making  environment  characterizing  the 
emerging  Operations  Control  Center  (OCC)  concept. 

Planned  work  by  CAMI  will  focus  on  the  extension 
and  elaboration  of  these  three  basic  concepts  —  secure 
and  efficient  knowledge  and  skill  tests,  simulation- 
based  assessment  of  abilities,  and  performance-  or 
simulation-based  assessments  of  team  work  —  to 
support  development  of  new  modules  to  add  to  the 
controller,  technician,  and  systems  specialist  selection 
processes.  Other  personnel  selection  technologies  such 


as  automated  training,  education,  and  experience  rat¬ 
ing  and  ranking  algorithms,  as  prototyped  in  FY  1996 
for  the  FG-0856  [Electronics  Technician]  occupa¬ 
tion,  will  also  continue  to  be  refined  and  extended. 

Implementation 

These  new  selection  technologies  will  be  developed 
within  a  framework  analogous  to  the  “plug  and  play” 
model  for  adding  hardware  to  a  personal  computer 
under  Windows  95®.  That  is,  new  test  modules  will 
be  developed  under  a  common  software  interface  so 
that  the  modules  can  be  easily  added  to  the  assessment 
system  to  reflect  new  or  changed  knowledge,  skill, 
ability,  or  performance  requirements  in  an  occupa¬ 
tion.  At  the  same  time,  new  measures  of  job  perfor¬ 
mance  would  be  developed  to  serve  as  validation 
criteria  for  the  new  tests.  Rather  than  mammoth, 
painful  validation  studies  involving  hundreds  of  con¬ 
trollers,  smaller,  more  focused  validation  studies  re¬ 
lating  specific  tests  to  specific  criteria  might  be 
conducted  using  sophisticated  statistical  techniques 
such  as  bootstrapping  and/or  resampling  where  tech¬ 
nically  and  legally  feasible.  The  focus  would  then  shift 
to  longer-term  evaluation  of  the  relationship  between 
predictors  and  job  performance  measures  over  time, 
to  identify  those  measures  that  are  most  efficient  and 
useful,  and  those  whose  predictive  utility  is  declining 
as  the  occupation  changes. 

This  approach  requires  the  creation  of  an  inte¬ 
grated  recruitment,  selection,  training,  and  job  per¬ 
formance  data  collection  and  analysis  system.  One 
possible  configuration  is  presented  in  Figure  3  as  an 
example  of  the  kinds  of  data  linkages  needed  to 
support  the  development  of  selection  systems  by  evolu¬ 
tion,  rather  than  revolution.  In  this  integrated  system, 
potential  recruitees  use  the  Remote  Vacancy  An¬ 
nouncement  for  Merit  Promotion  (REVAMP)  system 
to  obtain  information  about  job  vacancies  in  the  FAA, 
via  any  one  of  several  information  channels,  such  as 
the  Internet,  phone,  and  on-site  recruitment  visits. 
REVAMP  generates  an  application  package  as  re¬ 
quested.  The  applicant  then  submits  the  application 
to  the  Computerized  Applicant  Pools  (CAPS)  system 
for  initial  evaluation  and  screening.  CAPS  provides  an 
automated  rating  of  training,  education,  and  experi¬ 
ence,  and  ranks  candidates  relative  to  the  require- 
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ments  of  the  position  to  be  filled.  Qualified  applicants 
receive  the  appropriate  letter,  and  if  required,  notifi¬ 
cation  of  the  procedure,  date,  and  location  for  formal 
occupational  testing.  For  example,  an  applicant  for 
the  GS-2152  ATCS  occupation  might  be  screened  for 
basic  qualifications  under  CAPS  and  told  to  report  for 
further  testing  at  a  local  facility  or  contract  testing 
site,  such  as  a  junior  college.  Candidates  passing  any 
formal  tests  would  then  be  placed  on  a  centralized 
register  under  personnel  reform  for  referral  to  hiring 
authorities.  New  hires  would  then  be  scheduled  for 
required  training  (advanced  placement  examinations 
might  be  used  to  place  new  hires  into  “true  need” 
training  only).  From  there,  the  trained  personnel 
would  move  on  to  the  job. 

A  key  piece  of  this  infrastructure  would  be  a  data 
base  to  track  contacts,  application  data,  and  examinee 
scheduling  and  test  scores  for  equal  employment 
opportunity  (EEO),  validation,  cost,  and  other  man¬ 
agement  purposes.  Hiring,  training,  and  job  perfor¬ 
mance  data  would  also  be  tracked  in  the  data  base  to 
support  a  variety  of  data  analyses.  Analyses  might 
include  application  and  testing  validation  studies  to 
identify  revisions  to  employment  criteria.  They  could 
also  include  analyses  of  the  background,  ability,  train¬ 
ing,  education,  and  experience  profiles  of  successful 
candidates  at  each  stage  of  the  recruitment-selection¬ 
training-job  pipeline  to  develop  targeted  recruitment 
profiles.  Development  and  implementation  of  such  a 
comprehensive  feedback  system  is  absolutely  critical 
to  the  management  of  the  three  challenges  of  workforce 
generational  change,  technological  innovation,  and 
fiscal  constraints  after  the  turn  of  the  century. 

SUMMARY 

The  FAA  is  pursuing  a  three-phase  approach  to 
identifying  the  human  abilities  and  performance  re¬ 
quirements  of  the  emerging  NAS  architecuture  under 
the  Selection  and  Training  thrust  described  by  the 
1  5  National  Plan  for  Civil  Aviation  Human  Factors: 
An  Initiative  for  Research  and  Application  (FAA,  1 995). 
The  first  phase  is  to  develop  a  baseline  model  of  the 
jobs  of  our  technical  workforce  in  the  current  NAS 


architecture.  The  second  phase  is  to  develop  and  apply 
tools  for  identifying  changes  in  or  new  human  ability 
and  performance  requirements  in  parallel  with  ATC/ 
M  systems  development.  The  third  phase  is  to  de¬ 
velop,  validate,  and  deliver  innovative  personnel  se¬ 
lection  instruments  that  reflect  the  human  ability  and 
performance  requirements  of  the  future  NAS.  The 
program  is  designed  to  provide  agency  managers  with 
the  selection  tools  needed  to  assure  a  high  quality 
workforce  and  to  manage  personnel  costs,  genera¬ 
tional  change  in  the  technical  workforce,  and  techno¬ 
logical  innovation. 
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